Apparatus, method, program and recording medium for image restoration

ABSTRACT

An image of which restoration degree is determined based on the difference between each pixel and a neighboring pixel of the pixel in a light receiving element array  101  is divided into a plurality of areas based on the determined restoration degree, and the areas of the image are each restored in accordance with the determined restoration degree. By doing this, not only ringing and noise enhancement can effectively be suppressed but also the border between the areas which is conspicuous when the image is divided into two kinds of areas becomes less conspicuous, so that a natural restored image can be obtained.

[0001] This application is based on application No. 2001-100478 filed inJapan, the content of which is hereby incorporated by reference.

BACKGROUND OF THE INVENTION

[0002] 1. Field of the Invention

[0003] The present invention relates to an image restoration apparatus,an image restoration method, a program and a recording medium forrestoring the degradation of an image obtained as image data by adigital camera or the like.

[0004] 2. Description of the Related Art

[0005] Conventionally, various techniques have been proposed to restorethe degradation of an image obtained as image data by use of a lightreceiving element array such as CCD. Image degradation is that theactually obtained image is degraded compared to the ideal image to beobtained from the object to be taken. For example, an image obtained byuse of a digital camera is degraded by aberrations depending on, e.g.the aperture value, the focal length and the position of focus point,and is further degraded by an optical low pass filter for preventingspurious resolution. The image is also degraded by a camera-shake causedat the time of photo-taking.

[0006] On such a degraded image, restoration to approximate the obtainedimage to the ideal image by modeling image degradation hasconventionally been performed. For example, image restoration has beenperformed by, regarding image degradation as being caused while anoptical flux to be incident on each light receiving element is spreadingaccording to the Gaussian distribution, causing a restoration functionto act on the entire image or causing a filter that enhances an edge ofthe image to act on the entire image.

[0007] However, image degradation is not always caused on the entireimage. For example, when an object of a geometric pattern or with amonochromatic background is photographed or when an original forcharacter recognition is scanned, an area not affected by degradation ispresent in the image.

[0008] When image restoration is performed on the entire image, thereare cases where an area not requiring restoration is also adverselyaffected. For example, when restoration is performed on an area wherenoise and an edge are present, ringing and noise enhancement are caused,so that the area not requiring restoration is also adversely affected.

[0009] To solve this problem, a technique has already been proposed toperform image restoration while influence of ringing and the like isavoided by performing image processing only on a specific area.

[0010] However, according to this technique, since the area is dividedinto two kinds of areas, namely, an area to be restored and an area notto be restored, discontinuity on the border between the two kinds ofareas is conspicuous, so that the restored image is unnatural.

SUMMARY OF THE INVENTION

[0011] The present invention is made to solve the above-mentionedproblem, and an object thereof is to provide an image restorationapparatus and an image restoration method with which ringing and noiseenhancement are suppressed and a natural restored image can be obtained,and provide a program and a recording medium for causing a computer toperform such restoration.

[0012] To attain the above-mentioned object, one aspect of the presentinvention provides an image restoration apparatus comprising: arestoration degree determination part determining a restoration degreefor each of portions of an image; and a restoring part restoring each ofthe portions of the image in accordance with the determined restorationdegree by use of at least one point spread function representative of adegradation characteristic of the image.

[0013] According to this image restoration apparatus, the restorationdegree is determined for each of portions of the image, and each of theportions of the image is restored in accordance with the determinedrestoration degree. Consequently, not only ringing and noise enhancementwhich are caused when restoration is performed on areas not requiringrestoration are effectively suppressed but also the border between thedivisional areas is less conspicuous than when the image is divided intotwo kinds of areas, namely, an area on which restoration is performedand an area on which no restoration is performed. As a result, arestored image being natural as a while can be obtained.

[0014] Moreover, to attain the above-mentioned object, another aspect ofthe present invention provides an image restoration method comprising:determining a restoration degree for each of portions of an image; andrestoring each of the portions of the image in accordance with thedetermined restoration degree by use of at least one point spreadfunction representative of a degradation characteristic of the image.

[0015] According to this image restoration method, not only ringing andnoise enhancement are effectively suppressed but also discontinuity onthe border between the divisional areas is reduced unlike when the imageis divided into two kinds of areas, so that a natural restored image canbe obtained.

[0016] Moreover, to attain the above-mentioned object, further aspect ofthe present invention provides a program for causing a computer to carryout the following performance, and a recording medium on which thisprogram is recorded: determining a restoration degree for each ofportions of an image; and restoring each of the portions of the image inaccordance with the determined restoration degree by use of at least onepoint spread function representative of a degradation characteristic ofthe image.

[0017] According to this program and this recording medium, the computercan be caused to perform the processing to determine the restorationdegree for each of portions of the image and restore each of theportions of the image in accordance with the determined restorationdegree by use of at least one point spread function representative ofthe degradation characteristic of the image.

[0018] These and other objects, advantages and features of the inventionwill become apparent from the following description thereof taken inconjunction with the accompanying drawings, which illustrate specificembodiments of the invention.

BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In the following description, like parts are designated by likereference numbers throughout the several drawings.

[0020]FIG. 1 showing a first embodiment of the present invention is afront view of a digital camera;

[0021]FIG. 2 is a rear view of the digital camera;

[0022]FIG. 3 is a side view of the digital camera;

[0023]FIG. 4 is a longitudinal cross-sectional view of a lens unit andthe vicinity thereof in the digital camera;

[0024]FIG. 5 is a block diagram showing the internal structure of thedigital camera;

[0025]FIG. 6 is a view of assistance in explaining image deteriorationby the lens unit;

[0026]FIG. 7 is a view of assistance in explaining the imagedeterioration by the lens unit;

[0027]FIG. 8 is a view of assistance in explaining the imagedeterioration by the lens unit;

[0028]FIG. 9 is a view of assistance in explaining the imagedeterioration by the lens unit;

[0029]FIG. 10 is a view of assistance in explaining image degradation byan optical low pass filter;

[0030]FIG. 11 is a view of assistance in explaining the imagedegradation by the optical low pass filter;

[0031]FIG. 12 is a block diagram showing the structure of a functionassociated with photo-taking of the digital camera;

[0032]FIG. 13 is a flowchart of photo-taking by the digital camera;

[0033]FIG. 14 is a flowchart of a restoration routine;

[0034]FIG. 15 is a view of assistance in explaining an example ofsetting of a plurality of threshold values associated with the contrastof the obtained image;

[0035]FIG. 16 is a view of assistance in explaining an example ofsetting of continuous threshold values associated with the contrast ofthe obtained image;

[0036]FIG. 17 showing a second embodiment of the present invention is ablock diagram showing the structure of a function associated withphoto-taking of a digital camera;

[0037]FIG. 18 is a flowchart of a restoration routine;

[0038]FIG. 19 showing a third embodiment of the present invention is ablock diagram showing the structure of a function associated withphoto-taking of a digital camera; and

[0039]FIG. 20 is a flowchart of a restoration routine.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0040] FIGS. 1 to 4 show a first embodiment of the present invention.These figures show the appearance of a digital camera 1 as an imagerestoration apparatus. FIGS. 1 and 2 show a condition where a memorycard 91 is under insertion or removal. The memory card 91 is not shownin FIG. 3.

[0041] The external structure of the digital camera 1 is similar to thatof normal digital cameras. As shown in FIG. 1, on the front surface, alens unit 2 directing light from the object to the CCD and an electronicflash 11 emitting flash light toward the object are disposed. Above thelens unit 2, a viewfinder 12 for viewing the area to be photographed ofthe object is disposed.

[0042] On the top surface, a shutter start button 13 (hereinafter,referred to as start button) for starting photo-taking is disposed. Onthe left side surface, as shown in FIG. 3, a card slot 14 for insertingthe memory card 91 is provided.

[0043] On the rear surface of the digital camera 1, as shown in FIG. 2,the following are disposed: a liquid crystal display (LCD) 15 on whichan image obtained by photo-taking and an operation menu are displayed; achangeover switch 161 for switching between a photo-taking mode and areproduction mode; and four-way keys 162 accepting the user's selectioninput.

[0044] In FIG. 4, the lens unit 2 has a lens system 21 comprising aplurality of lens elements, and an aperture stop 22. In the rear of thelens unit 2, an optical low pass filter 31 and a color CCD 101 of onecharged coupled device system (hereinafter, referred to as CCD)comprising a two-dimensional light receiving element array are disposedin this order. That is, the lens system 21, the aperture stop 22 and theoptical low pass filter 31 constitute an optical system that directs thelight from the object to the CCD 101 in the digital camera 1.

[0045]FIG. 5 is a block diagram showing the principal internal structureof the digital camera 1. In FIG. 5, the start button 13, the changeoverswitch 161 and the four-way keys 162 are collectively shown as anoperation portion 16.

[0046] In FIG. 5, a CPU 41, a ROM 42 and a RAM 43 control the overalloperation of the digital camera 1, and are each connected to a bus. Bythe CPU 41 performing calculation and control according to an operatingprogram 421 stored in the ROM 42 with the RAM 43 as the work area, theoperation of each part of the digital camera 1 and image processing areperformed.

[0047] The lens unit 2 has not only the lens system 21 and the aperturestop 22 but also a lens driver 211 and an aperture stop driver 221 fordriving them. The lens system 21 and the aperture stop 22 are controlledas appropriate by the CPU 41 in accordance with the output of a distancemeasuring sensor and the brightness of the object.

[0048] The CCD 101 is connected to an analog to digital converter 33(hereinafter, referred to as A/D converter), and outputs the image ofthe object formed through the lens system 21, the aperture stop 22 andthe optical low pass filter 31 to the A/D converter 33 as an imagesignal. The image signal is converted to a data signal (hereinafter,referred to as image data) by the A/D converter 33, and is then storedinto an image memory 34. That is, the image of the object is obtained asimage data by the optical system, the CCD 101 and the A/D converter 33.

[0049] A corrector 44 performs various image processings such as whitebalance correction, gamma correction, noise removal, color correctionand color enhancement on the image data in the image memory 34. Thecorrected image data is transferred to a VRAM (video RAM) 151, wherebythe image is displayed on the display 15. When necessary, the image datais recorded onto the memory card 91 through the card slot 14 by theuser's operation.

[0050] In the digital camera 1, a processing to restore the degradationby an influence of the optical system is performed on the obtained imagedata. This restoration is realized by the CPU 41 performing calculationaccording to the operating program 421 stored in the ROM 42. While inthe digital camera 1, image processing (correction and restoration) issubstantially performed by processing image data, in the descriptionthat follows, the image data to be processed will be referred to simplyas “image” when necessary.

[0051] Next, image degradation in the digital camera 1 will bedescribed.

[0052] Image degradation is a phenomenon that the image obtained throughthe CCD 101, the A/D converter 33 and the like of the digital camera 1is not an ideal image. The image degradation is caused because a lightray emanating from one point on the object does not converge to onepoint on the CCD 101 but is distributed so as to spread. In other words,in a case where an ideal image is obtained, image degradation is causedbecause an optical flux to be incident on one light receiving element(that is, pixel) of the CCD 101 spreads to be incident on surroundinglight receiving elements.

[0053] In the digital camera 1 of the present embodiment, imagedeterioration by the optical system including the lens system 21, theaperture stop 22 and the optical low pass filter 31 is restored.

[0054] Reference number 71 of FIG. 6 represents the entire image. Whenit is assumed that in the case of an ideal image (that is, an image notdegraded by the image of the optical system; this image will hereinafterbe referred to as ideal image), the area designated by 701 is reproducedso as to be bright, in the case of the actually obtained image (referredto as obtained image), an area 711 larger than the area 701 isreproduced so as to be bright according to the focal length of the lenssystem 21 (corresponding to the moving out amount of a variator lens ina zoom lens system), the position of focus point and the aperture valueof the aperture stop 22. That is, the optical flux to be ideallyincident on an area on the CCD 101 corresponding to the area 701 spreadsto be actually incident on an area corresponding to the area 711.

[0055] When it is assumed that in the peripheral part of the image 71,in the case of an ideal image, the areas designated by 702 arereproduced so as to be bright, in the case of the obtained image, thesubstantially elliptical areas designated by 712 are reproduced so as tobe bright.

[0056] FIGS. 7 to 9 are schematic views of assistance in explaining theimage degradation by an optical influence of the lens unit 2 at thelevel of the light receiving elements of the CCD 101.

[0057]FIG. 7 shows a condition where an optical flux with an intensityof 1 is incident only on the light receiving element at the center of a3×3 light receiving element array when no degradation is caused by thelens unit 2 (that is, a condition where an ideal image is obtained). Onthe contrary, FIGS. 8 and 9 show conditions where the condition shown inFIG. 7 is changed by an influence of the lens unit 2.

[0058]FIG. 8 showing an example of the condition in the vicinity of thecenter of the CCD 101 shows a condition where light with an intensity of⅓ is incident on the central light receiving element and light with anintensity of ⅙ is incident on the upper, lower, right and left adjoininglight receiving elements. That is, FIG. 8 shows a condition where theoptical flux to be incident on the central light receiving elementspreads to be incident on the surrounding light receiving elements by aninfluence of the lens unit 2. FIG. 9 showing an example of the conditionin the peripheral part of the CCD 101 shows a condition where whilelight with an intensity of ¼ is incident on the central light receivingelement, light is incident while spreading from the upper left to thelower right.

[0059] This image degradation characteristic is called a point spreadfunction (or a point spread filter) because it can be expressed as afunction to convert the values of the pixels of an ideal image to thepixel value distribution shown in FIGS. 8 and 9 (that is, atwo-dimensional filter based on a point image distribution).

[0060] The point spread function representative of the characteristic ofthe degradation by an influence of the lens unit 2 can be obtained inadvance for each light receiving element position (that is, for eachpixel position) based on the focal length of the lens system 21, theposition of focus point and the aperture value of the aperture stop 22.Therefore, in the digital camera 1, as described later, the informationon the lens arrangement and the aperture value are obtained from thelens unit 2, the point spread function corresponding to the pixelposition is obtained, and restoration of the obtained image is realizedbased on the point spread function.

[0061] The point spread function associated with the lens unit 2 isgenerally a nonlinear function with the following as parameters: thefocal length; the position of focus point; the aperture value; and thetwo-dimensional coordinates on the CCD 101 (that is, of the pixels inthe image). While in FIGS. 7 to 9, no mention is made of the color ofthe image for convenience, in the case of a color image, a point spreadfunction corresponding to each of R, G and B or a point spread functionbeing a combination of the point spread functions of the colors isobtained. However, for simplification of the processing, the pointspread functions corresponding to R, G and B may be the same ignoringchromatic aberration.

[0062]FIG. 10 is a schematic view of assistance in explaining thedegradation by an influence of the optical low pass filter 31 at thelevel of the light receiving elements of the CCD 101. The optical lowpass filter 31 prevents spurious resolution by limiting the band by useof a birefringent optical material. In the case of a color CCD of onecharged coupled device system, the optical low pass filter 31 separatesthe light to be incident on the upper left light receiving element asshown in FIG. 10 into upper and lower parts as shown by the arrow 721,and then, separates the light into right and left parts as shown by thearrows 722.

[0063] In many cases, a green (G) filter is formed on two lightreceiving elements, on a diagonal line, of the four adjoining lightreceiving elements, and red (R) and blue (B) filters are formed on theremaining two light receiving elements in a color CCD of one chargedcoupled device system. The RGB values of the pixels are obtained byinterpolation with reference to the information obtained from thesurrounding pixels.

[0064] However, since the number of pixels of G is twice the numbers ofpixels of R and B in the color CCD of one charged coupled device system,when the data obtained from the CCD is used as it is, an image in whichthe resolution of G is higher than the resolutions of R and B isobtained, so that the high-frequency components of the object image thatcannot be captured by the light receiving elements on which R and Bfilters are formed appear as spurious resolution.

[0065] Therefore, the optical low pass filter 31 having a characteristicas shown in FIG. 10 is disposed on the front surface of the CCD 101.However, by an influence of the optical low pass filter 31, thehigh-frequency components of the image obtained from the light receivingelements of G are degraded.

[0066]FIG. 11 is a view illustrating the distribution of the opticalflux to be incident on the central light receiving element by theoptical low pass filter 31 having the characteristic shown in FIG. 10,that is, a view schematically showing the characteristic of the pointspread function corresponding to the optical low pass filter 31.

[0067] As shown in FIG. 11, the optical low pass filter 31 divides theoptical flux to be incident on the central light receiving element so asto be incident on 2×2 light receiving elements. Therefore, in thedigital camera 1, as described later, a point spread functioncorresponding to the optical low pass filter 31 is prepared, andrestoration of the obtained image is realized based on the point spreadfunction.

[0068] In the restoration using the point spread function associatedwith the optical low pass filter 31, the brightness component isobtained from the after-interpolation RGB values, and restoration isperformed on the brightness component. Moreover, as another restorationmethod, it may be performed to perform restoration on the G componentsafter the G components are interpolated and interpolate the R and Bcomponents by use of the restored G components.

[0069] While the point spread function is obtained for each pixel in thedescription given above, as the point spread function, a combination ofthe point spread functions of a plurality of pixels or a combination ofthe point spread functions of all the pixels, that is, a transformedstring corresponding to deterioration of a plurality of pixels may beobtained.

[0070] Next, image restoration will be described. Regarding the image asa vector in which pixel values are vertically arranged in raster orderand regarding the point spread function as a matrix, image restorationis frequently resolved into a problem of solving simultaneous linearequations. That is, when the restored image is X, the point spreadmatrix is H and the obtained image is Y, the relationship of thefollowing expression 1 is satisfied:

HX=Y  (1)

[0071] Generally, the obtained image Y includes noise, and the pointspread matrix H normally includes an observational error or adiscretization error. Therefore, a solution that minimizes theevaluation function E of the following expression 2 is obtained:

E=∥HX−Y∥ ²

min  (2)

[0072] To solve this minimization problem, using the fact that the pointspread matrix H is a large and sparse matrix, the iteration method isfrequently used.

[0073] However, when the iteration is repeated with the entire image,that is, all the components of the image X to be restored beingunknowns, noise such as ringing tends to be caused in the vicinity ofthe edge of the restored image. When the restoration is performed onareas where noise and an edge are present, ringing and noise enhancementare caused, so that areas not requiring restoration are also adverselyaffected.

[0074] Therefore, an example of a solution to this problem will bedescribed with reference to FIGS. 12 and 13.

[0075]FIG. 12 is a block diagram showing the structure of a functionassociated with photo-taking of the digital camera 1.

[0076] In FIG. 12, the digital camera 1 has the CCD 101, a point spreadfunction calculator 120, a point spread function storage 102, a restorer103, an area divider 104, a restoration degree determiner 105 and acorrector 106.

[0077] The CCD 101 as a light receiving element array converts theobject image to an image signal.

[0078] The point spread function calculator 120 obtains, at the time ofphoto-taking, the point spread function from the degradation informationsuch as the information on the lens arrangement and the aperture valuetransmitted from a lens control system and an aperture stop controlsystem. In the point spread function storage 102, the point spreadfunction calculated by the point spread function calculator 120 isstored.

[0079] The restoration degree determiner 105 determines the degree ofimage restoration based on the difference between each pixel and aneighboring pixel of the pixel in the image obtained by the CCD 101, forexample, the difference in contrast.

[0080] The area divider 104 divides the image into a plurality of areasbased on the determined restoration degree. The restorer 103 restoresthe area, other than the area not to be restored, of the image inaccordance with the determined restoration degree by use of at least onepoint spread function representative of the degradation characteristicof the image. The corrector 106 makes a gradation correction and thelike on the restored image.

[0081] Next, photo-taking by the digital camera 1 will be described withreference to the flowchart of FIG. 13. In the description that followsand the figures, steps are abbreviated to S.

[0082] In FIG. 13, when the start button 13 is depressed, at S101, theoptical system is controlled in order to form an image of the object onthe CCD 101. That is, a lens controller (not shown) supplies a controlsignal to the lens driver 211, whereby the distances between the lenselements included in the lens system 21 are controlled. Further, acontrol signal is applied from an aperture stop controller (not shown)to the aperture stop driver 221 to control the aperture stop 22.

[0083] At S102, the information on the lens arrangement and the aperturevalue are transmitted from the lens controller and the aperture stopcontroller to the point spread function calculator 120 as degradationinformation for obtaining the point spread function. Then, at S103,exposure is performed, and the image of the object obtained by the CCD101 and the like is stored into the image memory 34 as image data. Thesucceeding image processing is performed on the image data stored in theimage memory 34.

[0084] At S104, the point spread function calculator 120 obtains thepoint spread function of each pixel considering influences of the lenssystem 21 and the aperture stop 22 by use of the degradation informationsupplied from the lens controller and the aperture stop controller. Theobtained point spread function is stored in the point spread functionstorage 102. In the point spread function storage 102, the point spreadfunction associated with the optical low pass filter 31 is stored inadvance.

[0085] At S104, it may be performed to separately obtain the pointspread function for each structure and characteristic of the lens unit 2and then, obtain the point spread function considering the entireoptical system. For example, the point spread function associated withthe lens system 21, the point spread function associated with theaperture stop 22 and the point spread function associated with theoptical low pass filter 31 may separately be stored. Further, the pointspread function associated with the lens system 21 may also separatelybe obtained as the point spread function associated with the focallength and the point spread function associated with the position offocus point.

[0086] Moreover, to simplify the calculation to obtain the point spreadfunction of each pixel, it may be performed to obtain the point spreadfunction of a representative pixel in the image and then, obtain thepoint spread functions of the other pixels by interpolating the pointspread function of the representative pixel.

[0087] After the point spread function is obtained, at S105, therestorer 103 performs restoration on a predetermined divisional area ofthe obtained image (degraded image). Consequently, the degradation inthe obtained image by an influence of the optical system is removed fromthe image. This restoration will be described later.

[0088] After the restoration, at S106, the corrector 106 performsvarious image processings such as white balance correction, gammacorrection, noise removal, color correction and color enhancement on therestored image, and at S107, the corrected image data is stored in theimage memory 34. The image data in the image memory 34 is stored intothe memory card 91 through the card slot 14 as required.

[0089] The subroutine of the image restoration (S105) is shown in FIG.14.

[0090] First, at S11, a plurality of threshold values associated withthe contrast of the obtained image is calculated, and the restorationlevel (degree) is determined.

[0091] Here, contrast is the difference between the values of the targetpixel and a neighboring pixel, and can be obtained by convoluting theobtained image and an appropriate differential filter and calculatingthe absolute value for each pixel (hereinafter, this image will bereferred to as gradient image). That is, a threshold value is set to thegradient value and the restoration degree V is determined.

[0092] For example, five threshold values t1 to t5 are set as shown inFIG. 15, and the restoration degree V is set for each of the followingsix levels: 0 to t1; t1 to t2; t2 to t3; t3 to t4; t4 to t5; and higherthan t5. Here, V=0 indicates that no restoration is performed, and V=1indicates that the image is completely restored. The other fourintermediate values are considered to indicate the restoration degreescorresponding to the values.

[0093] While the restoration degree V is set for the six gradient levelsin the description given above, the restoration degree V may be set forall the gradient values as shown in FIG. 16.

[0094] Then, at S12, the obtained image is divided into a plurality ofareas by use of the restoration degree V. That is, by determining whichof the values of the six restoration degrees V each pixel belongs to byuse of the gradient image, the entire image is divided into six classes.In the description that follows, the diagonal element represents thevalue of the restoration degree V, and the diagonal matrix having anelement whose position in the matrix corresponds to the pixel number andhaving an element being the square of the number of pixels is newlyrepresented by the restoration degree V.

[0095] After the area division, at S13, the area to be restored of theimage is restored by use of the point spread function in accordance withthe determined restoration degree V. This is equivalent to solving aminimization problem (P), with an equal sign constraint condition, ofthe following expression 3:

∥V^(1/2)(HX−Y)∥²

min   (P) . . . . . . (3)

[0096] s.t. X_(i)=X_(i) ^(O)(for i∈{j|V_(jj)=0})

[0097] where V^(1/2) is a diagonal matrix having the square root of thediagonal element of V as the diagonal element, X⁰ is the degradedobtained image, and

X_(i) ⁰  (4)

[0098] is the i component thereof.

[0099] It is considered that in the problem (P), for the pixels wherethe element of the weighting matrix V representative of the restorationdegree is 0, the pixel values of the obtained image Y are used as theyare, and for the other pixels, weight is assigned to the residualbetween the pixels in accordance with the value of V.

[0100] Expanding the norm, the problem (P) is equivalent to a quadraticprogramming problem (P′) of the following expression 5:

X^(T)H^(T)VHX−2X^(T)H^(T)V

min   (P′) . . . (5)

[0101] s.t. X^(i)=X⁰ _(i)(for i∈{j|V_(jj) =0})

[0102] The quadratic programming problem (P′) can be solved, forexample, by use of an iteration method such as the steepest-descentmethod (Richardson method) with a fixed step size like the followingexpression 6:

X _(i) ^(n+1) =X _(i) ^(n) −k(H ^(T) V(HX ^(n) −Y))_(i) foriε{j|V_(jj)≠0}

X _(i) ^(n+1) =X _(i) ⁰ for iε{j|V_(jj)=0}

[0103] where

X_(i) ^(n)  (7)

[0104] is the i-th component of the updated image X after the n-thiteration, and k is an appropriately selected constant.

[0105] Convergence is determined based on whether a residual norm ∥HX−Y∥or a weighted residual norm ∥V^(1/2)(HX−Y)∥ is smaller than apredetermined value or not.

[0106] As described above, in the digital camera 1, since the imagedegradation by an influence of the optical system is removed by use ofthe point spread function representative of the characteristic of thedegradation by the optical system, an influence such as occurrence ofringing and increase in noise on areas not affected by degradation canbe suppressed, and the image is divided into a plurality of areas basedon the restoration degree V and each of the areas of the image isrestored in accordance with the determined restoration degree V,discontinuity on the border between a plurality of divisional areas isreduced unlike when the image is merely divided into two areas, so thata natural restored image can be obtained.

[0107] Next, a second embodiment of the present invention will bedescribed.

[0108] When an area to be restored is determined by use of the contrastof the obtained image Y, the contrast is low in a completely blackenedarea, and this area is not included in the area to be restored. That is,the point spread function has a characteristic that eliminates orreduces a specific frequency component, and for example, there are caseswhere the contrast of a striped area in the original (ideal image)becomes substantially zero in the obtained image Y.

[0109] An apparatus and a method modifying, to solve this problem, imagedivision based on the contrast of the obtained image (degraded image)and the contrast of the restored image will be described with referenceto FIGS. 17 and 18.

[0110]FIG. 17 is a block diagram showing the structure of a functionassociated with photo-taking of the digital camera 1. The same parts asthose of FIG. 12 are designated by the same reference numbers anddescription thereof is omitted.

[0111] In FIG. 17, the digital camera 1 has first and second restorationdegree determiners 105A and 105B, first and second area dividers 104Aand 104B and an area modifier 107.

[0112] The first restoration degree determiner 105A determines the imagerestoration degree based on the difference in contrast between eachpixel and a neighboring pixel of the pixel in the image obtained by theCCD 101. The first area divider 104A divides the image into a pluralityof areas based on the determined restoration degree. The restorer 103restores the area, other than the area not to be restored, of the imagein accordance with the determined restoration degree like in the firstembodiment.

[0113] The second restoration degree determiner 105B re-determines therestoration degree based on the restored image (the contrast of therestored image) and the obtained image (the contrast of the obtainedimage). The second area divider 104B re-divides the image into aplurality of areas based on the restoration degree determined by thesecond restoration degree determiner 105B. The area modifier 107modifies the area to be restored to re-perform the image restoration bythe restorer 103.

[0114] The subroutine of the image restoration (S 105) in the secondembodiment is shown in FIG. 18.

[0115] First, at S21, the first restoration degree determiner 105Acalculates a plurality of threshold values associated with the contrastof the obtained image, and determines the restoration degree. The methodof the determination is the same as that of the first embodiment; athreshold value is set to the gradient value and the restoration degreeis determined.

[0116] Then, at S22, the first area divider 104A divides the obtainedimage into a plurality of areas by use of the determined restorationdegree V. The method of the division is also the same as that of thefirst embodiment; by determining which of the values of the sixrestoration degrees V each pixel belongs to by use of the gradientimage, the entire image is divided into six classes. After the areadivision, at S23, the area to be restored of the image is restored byuse of the point spread function in accordance with the determinedrestoration degree V.

[0117] At S24, after the restored image is obtained, the secondrestoration degree determiner 105B re-determines the restoration degreebased on the contrast of the obtained image and the contrast of therestored image, the second area divider 104B re-divides the restoredimage based on the re-determined restoration degree, and the areamodifier 107 modifies the area to be restored.

[0118] Assume now that the restoration degree weighting matrix based onthe contrast of the obtained image Y is Vo and the restoration degreeweighting matrix based on the contrast of the restored image is Vr. Forexample, when the entire image is the area to be restored, Vo is anidentity matrix.

[0119] When a matrix in which the elements of the restoration degreeweighting matrix Vr that are lower than a given threshold value are zerois Vr′ and a matrix in which the elements of the restoration degreeweighting matrix Vo that correspond to the remaining elements of therestoration degree weighting matrix Vr are zero is Vo′, a newrestoration degree weighting matrix Vn can be calculated by thefollowing expression 8:

V _(n) =V _(a) ′+V _(r)′  (8)

[0120] Then, at S25, restoration is performed by use of the newrestoration degree weighting matrix Vn.

[0121] Since area modification is performed by use of the restored imageas described above, restoration accuracy is further improved.

[0122] Next, a third embodiment of the present invention will bedescribed.

[0123] Normally, random noise is superimposed on the obtained image andthe point spread function, and there are cases where the noise isenhanced in the restored image. An apparatus and a method composing aplurality of restored images restored by use of different restorationdegree weighting matrices to solve this problem will be described withreference to FIGS. 19 and 20.

[0124]FIG. 19 is a block diagram showing the structure of a functionassociated with photo-taking of the digital camera 1. The same parts asthose of FIG. 17 are designated by the same reference numbers anddescription thereof is omitted.

[0125] In FIG. 19, the digital camera 1 has a composer 109 as well asthe first and the second area dividers 104A and 104B and the first andthe second restoration degree determiners 105A and 105B, and therestorer 103 has a first restorer 103A, a second restorer 103B and astorage 103C.

[0126] The first restorer 103A restores the area, other than the areanot to be restored, of the image in accordance with the restorationdegree determined by the first restoration degree determiner 105A. Thesecond restorer 103B restores the area, other than the area not to berestored, of the image in accordance with the restoration degreedetermined by the second restoration degree determiner 105B. In thestorage 103C, the image restored by the first restorer 103A istemporarily stored. The composer 109 composes the first restored imagestored in the storage 103C and the second restored image restored by thesecond restorer 103B.

[0127] The subroutine of the image restoration (S105) in the thirdembodiment is shown in FIG. 20.

[0128] Here, a case where two restored images are used will be describedas an example.

[0129] At S31, the image is restored based on a restoration degreeweighting matrix V1, and the image is designated by I₁. At S32, therestored image I₁ is temporarily stored in the storage 103C.

[0130] Then, at S33, the image is restored based on another restorationdegree weighting matrix V₂, and the image is designated by 12. At S34,the composer 109 composes the restored image I₁ and the restored imageI₂ by use of an appropriate weight V (0≦V_(i)≦1) by the followingexpression 9:

I=V·I ₁+(1−V)I ₂  (9)

[0131] By composing a plurality of images as described above, noise isreduced in the composite image I.

[0132] While the image restoration is performed by the digital camera 1in the description given above, an image taken by a digital camera orthe like may be restored by an external apparatus such as a computer. Inthis case, the restoration routine described in each embodiment is aprogram. The program is read into a computer through a network or arecording medium, and is executed on the computer.

[0133] As described above, by determining the restoration degree foreach portion of the image and restoring each portion of the imageaccording to the determined restoration degree, ringing and noiseenhancement are suppressed to the utmost, and discontinuity on theborder between the divisional areas is reduced unlike when the image isdivided into two kinds of areas, so that a natural restored image can beobtained.

[0134] Moreover, by the restoration degree determination partdetermining the restoration degree of each portion based on thedifference between each pixel and a neighboring pixel of the pixel, therestoration degree can accurately be determined.

[0135] Further, by providing the second restoration degree determinationpart for re-determining the restoration degree of each portion based onthe image restored by the restoring part and the second restoring partfor re-restoring the image in accordance with the re-determinedrestoration degree, for example, even when the contrast is so low thatthe area is not determined to be an area to be restored, since the areais modified in the restored image and restoration is performed,restoration can highly accurately be performed.

[0136] Further, by providing the composing part for composing the imagerestored by the restoring part and the image restored by the secondrestoring part and composing the images of different restorationdegrees, for example, noise can effectively be prevented from beingenhanced in the restored image.

[0137] The above-described restoration may be performed by a computer.

[0138] Although the present invention has been fully described by way ofexamples with reference to the accompanying drawings, it is to be notedthat various changes and modifications will be apparent to those skilledin the art. Therefore, unless otherwise such changes and modificationsdepart from the scope of the present invention, they should be construedas being included therein.

What is claimed is:
 1. An image restoration apparatus comprising: arestoration degree determination part determining a restoration degreefor each of portions of an image; and a restoring part restoring each ofthe portions of the image in accordance with the determined restorationdegree by use of at least one point spread function representative of adegradation characteristic of the image.
 2. An image restorationapparatus according to claim 1, wherein the restoration degreedetermination part determines the restoration degree of each portionbased on a difference between each pixel and a neighboring pixel of thepixel.
 3. An image restoration apparatus according to claim 1, furthercomprising: a second restoration degree determination partre-determining the restoration degree of each portion based on an imagerestored by the restoring part; and a second restoring part re-restoringthe image based on the re-determined restoration degree.
 4. An imagerestoration apparatus according to claim 3, further comprising acomposer composing the image restored by the restoring part and an imagerestored by the second restoring part.
 5. An image restoration methodcomprising: determining a restoration degree for each of portions of animage; and restoring each of the portions of the image in accordancewith the determined restoration degree by use of at least one pointspread function representative of a degradation characteristic of theimage.
 6. An image restoration method according to claim 5, wherein inthe determination of the restoration degree, the restoration degree ofeach portion is determined based on a difference between each pixel anda neighboring pixel of the pixel.
 7. An image restoration methodaccording to claim 5, further comprising: re-determining the restorationdegree of each portion based on an image restored by the restoring part;and re-restoring the image in accordance with the re-determinedrestoration degree.
 8. An image restoration method according to claim 7,further comprising composing the restored image and a re-restored image.9. A computer-readable program based on which a computer performs thefollowing operations: determining a restoration degree for each ofportions of an image; and restoring each of the portions of the image inaccordance with the determined restoration degree by use of at least onepoint spread function representative of a degradation characteristic ofthe image.
 10. A computer-readable program according to claim 9, whereinin the determination of the restoration degree, the restoration degreeof each portion is determined based on a difference between each pixeland a neighboring pixel of the pixel.
 11. A computer-readable programaccording to claim 9, further comprising: re-determining the restorationdegree of each portion based on an image restored by the restoring part;and re-restoring the image in accordance with the re-determinedrestoration degree.
 12. A computer-readable program according to claim11, further comprising composing the restored image and a re-restoredimage.
 13. A recording medium in which a program based on which acomputer performs the following operations is stored: determining arestoration degree for each of portions of an image; and restoring eachof the portions of the image in accordance with the determinedrestoration degree by use of at least one point spread functionrepresentative of a degradation characteristic of the image.
 14. Arecording medium according to claim 13, wherein in the determination ofthe restoration degree, based on the program, the computer determinesthe restoration degree of each portion based on a difference betweeneach pixel and a neighboring pixel of the pixel.
 15. A recording mediumaccording to claim 13, wherein based on the program, the computerfurther re-determines the restoration degree of each portion based on animage restored by the restoring part, and re-restores the image inaccordance with the re-determined restoration degree.
 16. A recordingmedium according to claim 15, wherein based on the program, the computerfurther composes the restored image and a re-restored image.